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Summary of Review 

The School Choice Demonstration Project has published a series of reports written in the fifth 
and final year of its evaluation of the Milwaukee Parental Choice Program (MPCP) . This review 
is of Report #30, a final follow up to a five-year study examining high school graduation and 
post- secondary enrollment rates for students participating in the MPCP. Researchers tracked an 
initial sample of MPCP students enrolled in either 8th or 9th grade in 2006 and compared their 
high school graduation and college enrollment rates with a sample of Milwaukee Public School 
(MPS) students. The report found that voucher students who attended a private school in 8th or 
9th grade in 2006 “were more likely to graduate high school,” “enroll in a four-year post- 
secondary institution,” and “persist in that four-year institution beyond the first year of 
enrollment.” Such conclusions should be considered alongside at least two important caveats, 
however. The first is a methodological concern. Roughly 75% of the original sample of 80 1 
MPCP 9th graders were not still enrolled in a MPCP high school in 12th grade. The inferences 
drawn about the effects of the MPCP on graduation rates compared with those in the MPS are 
severely clouded by substantial sample attrition. A second concern lies in the report’s 
interpretation of the data. Among the most careful statistically controlled analyses, only one 
finding was statistically significant at conventional levels. These two limitations prevent broad 
conclusions being drawn about the relative effectiveness of the MPCP and the MPS on 
graduation and higher education continuation rates. 
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Review Of 
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I. Introduction 

The Milwaukee Parental Choice Program (MPCP) is currently the largest urban school 
voucher program in the nation. At present, nearly 21,000 students use a voucher of up to 
$6,442 to attend secular or religious private schools in Milwaukee. Voucher programs have 
long been the objects of intense debates over the efficient and appropriate use of public 
education funds. 

In February 2012 the School Choice Demonstration Project 1 released a series of final 
reports from its five-year evaluation of the MPCP. The Wisconsin Legislature in 2005 
required MPCP schools to administer nationally normed tests in grades 4, 8, and 10 to 
MPCP students, and to also submit the test scores to the School Choice Demonstration 
Project for puiposes of evaluation. The reports address a range of issues related to the 
MPCP, including special education services, school climate and contexts, and test score 
performance. Of particular interest to policymakers has been the performance of MPCP 
students relative to students in the Milwaukee Public Schools (MPS). The School Choice 
Demonstration Project report under review here is Report # 30 , Student Attainment and 
the Milwaukee Parental Choice Program: Final Follow-up Analysis, which addresses high 
school graduation and post- secondary enrollment rates of the voucher program in 
comparison with traditional public school students. It is authored by J oshua M. Cowen, 
David J . Fleming, J ohn F. Witte, Patrick J . Wolf and Brian Kisida. 2 

II. Findings and Conclusions of the Report 

This report is a final follow-up to a five-year study examining high school graduation and 
post- secondary enrollment rates for students participating in the MPCP. Researchers 
tracked an initial sample of MPCP students in either 8 th or 9 th grade in 2006 and compared 
their high school graduation and college enrollment rates with those of a sample of MPS 
students. The report found that voucher students who attended a private school in 8 th or 
9 th grade in 2006 “were more likely to graduate high school,” “enroll in a four- year post- 
secondary institution,” and “persist in that four- year institution beyond the first year of 
enrollment” (p. 16). 
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The report found that most MPCP and MPS students who entered post- secondary 
institutions attended University of Wisconsin- Milwaukee, the University of Wisconsin- 
Whitewater, or Alvemo College. Among college- goers in both sectors, MPCP students were 
more likely to attend religious or private four- year institutions, more likely to enroll in 
slightly more expensive institutions, and more likely to attend institutions whose students 
had slightly lower SAT and ACT scores. There were no differences in acceptance rates 
among institutions attended by MPCP and MPS students. 

III. The Report’s Rationale for Its Findings and Conclusions 

To the report’s credit, it offered several caveats to its findings. The first entailed the 
relative smallness of the MPCP high school program. The panel of 801 9 th graders under 
examination constituted the entire population of MPCP 9 th graders in 2006, representing 
less than 5% of all MPCP students that year. Moreover less than one- fourth of all MPCP 
schools served high school grades. The report raised the possibility that such “small 
numbers could exacerbate [any] selection bias problems” (p. 16). To examine this prospect, 
MPCP 8 th graders enrolled in 2006 were compared with students who remained in MPCP 
the following year (9 th grade); the comparison revealed “no systematic evidence that those 
students who remain in the MPCP for 9 th grade are dramatically different in terms of 
demographics, prior achievement or these other demographic measures from MPCP 
students who switch to the MPS for high school” (p. 16). 

A second caveat noted by the report is critically important to evaluating the validity of the 
inferences it draws from its findings. The report indicates that the “majority of students 
(approximately 75 percent)” who were enrolled in the 9 th grade in MPCP were not enrolled 
there by the time they reached 12 th grade” (p. 16). The report acknowledges that the 
“results of this paper as a whole should therefore be interpreted as the effect of ‘exposure’ 
to the MPCP rather than long-term persistence in that sector” (p. 16). No analyses were 
offered that examined the relationship between MPCP “exposure” and educational 
attainment. Readers are made aware that roughly 600 of the original 801 MPCP 9 th 
graders were not enrolled in an MPCP high school in 12 th grade; however, there is no 
indication of when students left (most likely because researchers did not have access to 
this information). The report acknowledges that “students who leave MPCP for public 
schools are among the lowest performing private school students” (p. 16). 

Finally, a third caveat is that this type of analysis examines overall differences on average 
between the two sectors and is not designed to investigate what individual schools might 
be doing to cause such differences. The report references companion qualitative 
examinations better suited for this task (e.g.. Report 34: School Site Visits: What Can We 
Learn from Choice Schools in Milwaukee?) . 

The report aptly concludes that even “if the results we present here are interpreted as 
evidence that MPCP students are performing slightly better on one metric— attaining a 
given level of education— they do not support a comprehensive conclusion that the MPCP 
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necessarily provides a better learning environment than MPS” (p. 17, emphasis in 
original) . 



IV. The Report’s Use of Research Literature 

The report references several peer-reviewed studies that are relevant to the research. In 
particular it is very thorough in its summary of research on a variety of life outcomes 
associated with higher educational attainment. 

V. Review of the Report’s Methods 

Three aspects of the report’s methods are highlighted here: sample matching techniques, 
calculation of graduation rates and continuation rates. 

Sample Matching Techniques 

The sampling procedures are described in sufficient detail, particularly the process by 
which the sample of MPS students was matched to the MPCP sample. The primary MPCP 
sample (n=801) was actually the entire population of 9 th grade MPCP students enrolled in 
2006-07. This group was supplemented by a “refreshed” sample of 290 MPCP 8 th graders 
enrolled the same year and was drawn from “grade- stratified representative samples.” 

The report aptly underscores the importance of establishing an MPS sample that is 
comparable to the MPCP group, particularly on characteristics that affect student 
achievement. In this case, the researchers matched each MPCP student to an MPS student 
on observable characteristics, including race, gender, and prior test scores. 

When possible, the report used census tract as one of the observable characteristics, 
matching MPCP students with an MPS student with similar demographic and academic 
characteristics located in the same neighborhood. 

Graduation Rate Determination 

Determining who among the 1,091 8 th and 9 th graders in 2006 graduated four and five 
years later was complicated by students switching schools, leaving the state, dropping out, 
or otherwise disappearing from state graduation lists. The report describes a process of 
attempting to locate student names on lists of MPS and MPCP graduates from the 2009- 
2010 and 2010-2011 school years. The report does not indicate how many students were 
not located, and simply states. 

These sources, while valuable for confirming graduation status and current 
enrollment, did not provide us with all information needed for our analysis. In 
particular, we could not determine students who may have graduated from 
schools outside of either MPCP or MPS (p. 5). 
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The report also acknowledges that “[a] student who began our study in the MPCP panel 
could have graduated from MPS, and vice-versa. The operation of the school choice 
program, specifically the scarcity of high schools in the program, makes MPCP- to- MPS 
transfers particularly common (Cowen et al. 2010)” (p. 5). So some students who began in 
MPCP later graduated from MPS (and apparently vice versa) but the length of time in 
either program was not factored in the analysis. Notably, more than half the students 
(56%) in the MPCP 9 th grade sample were not in the MPCP four years later. Curiously, it 
fails to state how many program- switchers there were, when they switched and in which 
direction, and how many graduated. I will return to this point later, as it is critically 
important to any inferences made of the findings. 

The authors did note that phone surveys in 2010 and 2011 “increased the response of the 
original 9 th grade sample to nearly 75 percent” (p. 5). Similar overall response rates were 
not provided for the 8 th grade refresh sample. 

Graduation rates among the MPCP and MPS samples were determined by dividing the 
total number of graduates in each group — irrespective of which sector a student 
graduated from — by the number of students known to have graduated. Graduation results 
are presented in the report’s Table 1 (p. 6). To its credit, the report does not stop at this 
point. It conducts more sophisticated analyses to help account for possible systematic 
differences between students in the two sectors, statistically controlling for the influence 
of race, gender, and academic ability on graduation rates. These additional controls 
present a stronger argument for any difference in graduation rates between sectors. But, in 
fact, the results (presented in the report’s Table 2, p. 8), indicate that there is no 
appreciable difference in graduation rates between the two sectors at conventional levels 
of statistical probability. The report denotes this finding as significant at the p<. 10 level, 
which is technically accurate but can be somewhat misleading. Reporting statistical 
significance at this level is not unheard of, but it is also a more lenient criterion than is 
usually acceptable in social science research— particularly when samples of this size, 
where statistical significance is easier to obtain, are analyzed. 

Post-secondary Continuation Rates 

The analyses conducted for post- secondary enrollment were performed in similar fashion. 
Initial summary statistics were computed by sector for students attending two- and four- 
year colleges, and then more sophisticated multivariate analyses were employed. For these 
analyses, in addition to controlling for race, gender, and prior math and reading 
achievement, the authors adjusted for parental characteristics, such as education level and 
income. The researchers should be commended for adding these variables to the model, as 
parental socioeconomic status is a strong predictor of student academic and attainment 
outcomes. After all of these student characteristics were taken into account, the effects of 
being in the voucher program were not statistically significant at conventional levels (see 
the report’s Table 9, p. 15). The report again relies on the far looser statistical probability 
levels of .10 in summarizing these results, reporting that “only the high school graduate and 
four-year enrollment differences remain statistically significant at a level approaching convention 
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(p<0.10, two-tailed)” (p. 14). After offering caveats and cautions, the report concludes: 
“Nevertheless, the results in Table 9 generally confirm the positive MPCP difference in 
educational attainment, even after adjusting in particular for the attainment level of a 
student’s parent” (p. 14). 

VI. Review of the Validity of the Findings and Conclusions 

Summary statements found in the report’s executive summary and conclusion, while not 
inaccurate, do invite conclusions about MPCP effects that are likely not warranted by the 
data presented. For example, the report concludes that the “results here suggest that 
students who used a voucher to attend private school in 8 th or 9 th grade were more likely to 
graduate high school” (p. 16). The problem is that we do not know exactly where they 
graduated high school or for how long they were enrolled in a voucher program school. 

This one caveat alone calls into question the usefulness of nearly the entire study. 

Several of the report’s summary statements are not necessarily supported by the analyses. 
A set of analyses were conducted that compared MPCP and MPS attainment; each analysis 
was more rigorous or tightly controlled than the previous one. The final set of analyses 
indicated that differences in attainment rates between MPCP and MPS sectors were 
negligible. Indeed, among the more carefully statistically controlled analyses only one 
finding was statistically significant at conventional levels: the report’s Table 5 results 
controlled for prior achievement and showed statistically significant differences (p<.01) on 
attending and persisting in four- year post- secondary institutions. The report’s findings 
may be much ado about nothing. 

VII. Usefulness of the Report for Guidance of Policy and Practice 

The report shines light on the important indicators of educational attainment, such as high 
school graduation and enrollment in college. Such indicators are often over- shadowed by 
test performance, which is heavily emphasized by accountability systems and politicians, 
and ubiquitous as dependent variables in research examining the efficacy of school choice 
programs. Educational attainment indicators, in contrast, are far more relevant and 
important to students and their families. 

A significant point to note about this report is that there really aren’t many, if any, 
differences to report here. Even if there were, the research design is not robust enough to 
inform the reader about the causal effects of a voucher program. 
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Notes and References 



1 The School Choice Demonstration Project (SCDP) is housed in the Department of Education Reform at the 
University of Arkansas (http:/ / www.uark.edu/ ua / der/ SCDP/ Research.html) . 

2 Cowen, J. M., Fleming, D. J., Witte, J. F., Wolf, P. J., &Kisida. B. (2012). Student Attainment and the Milwaukee 
Parental Choice Program: Final Follow-up Analysis. Fayetteville: University of Arkansas School Choice 
Demonstration Project. Retrieved Februaiy 27, 2012, from 

http:/ / www.uaedreform.org/ SCDP/ Milwaukee_ Research.html. 
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